Imaging devices, such as digital cameras, may perform phase detection auto focus (AF) in order to determine a lens shift value used to shift a current lens position such that a captured image of a scene will be in focus. Phase detection AF may include the measurement of a phase difference between specialized phase detection pixels. The accuracy of the phase detection measurement may depend on various visual properties of the scene. Accordingly, a confidence level may be calculate for each phase detection measurement which is indicative of the likelihood that the phase detection measurement is accurate. In this context, there is a need for improvement in the accuracy of phase detection AF based on the measured phase difference and the corresponding confidence level.